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CROSS REFERENCE TO RELATED APPLICATIONS 
(0001] Pursuant to 35 U.S.C. § 1 19(a), this application claims the benefit of earlier 
filing date and right of priority to the Korean Patent Application No. 2002-80469, filed on 
December 16, 2002 s the content of which is hereby incorporated by reference in its entirety. 

15 

BACKGROUND OF THE INVENTION 

Field of the Invention 

[0002] The present invention relates to an apparatus for operating a mobile 
communication terminal with an integrated photographic apparatus and method thereof, and 
20 more particularly, to an apparatus for using a photographic image to operate a mobile 
communication terminal with an integrated photographic apparatus and method thereof. 

Description of Related Art 

[0003] In general, mobile communication terminals include cellular phones or 
25 Personal Communication Systems (PCS) having wireless communication capabilities. As the 
need for video/image communication emerges along with voice communication, several types of 
mobile communication terminals with an integrated photographic apparatus have been suggested 
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to satisfy this demand. Furthermore, such terminals have become increasingly complex and 
incorporate many features and functions which a user may find extremely difficult and 

complicated to operate. 

[0004] Referring to FTGs. 1A and IB, operative methods for a conventional mobile 

5 telecommunication terminal, exemplified in the fomi of a folding-type mobile telephone, are 
shown. In FIG. 1 A, the user, exemplified by the hand, operates the mobile terminal by 
interacting with one or more of the terminal manipulation devices. In the example shown in FIG. 
1 A, such devices are in the form of a keypad having buttons. Due to the multiplicity functions 
and features in mobile terminals, each individual button is directed to a variety of functions. For 

1 0 example, a single button may represent 10 different characters when the mobile terminal is used 
for text messaging purposes, 

[0005] Alternatively, the single button may represent a "short cut' 1 key to operate a 
function of the terminal in quickened manner, such as taking a photographic with the integrated 
photographic apparatus. Ultimately, the number of functions attributed to each terminal 

15 manipulation device, which is finite, becomes limited due to acceptable ergonomics in using the 
mobile terminal . Attributing too many functions to a finite number of keys may result in the 
terminal losing overall effectiveness and popularity with respect to user-friendliness. 
Furthermore, as terminals technologically evolve and decrease in size, repetitive engagement of 
the terminal manipulation devices, which become increasingly smaller, a user will find operation 

20 of the terminal more difficulty and tedious. 

[0006] In PIG. 1 8, a user, exemplified by the face, operates the mobile terminal via 
voice command, in which the user verbally communicates a command for the terminal to operate 
a certain function. In this example, the user states the term, "MY HOUSE," in order for the 
terminal to receive, process, and function in the form of dialing the user's residence. Currently, 
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many voice commands are relegated to performing simple functions, such as dialing a telephone 
number. Furthermore, audibly distinct terms must be used for each stored number, thereby 
requiring the user to memorize such terms. This can become difficult when a specific location, 
such as a user's residence, has more than one telephone number associated with. Also, the 
5 number of voice commands is limited to the internal memory of the terminal, which is finite and 
likely decreased as the user continuously adds other functions to the terminal. 

(0007] Therefore, an improved solution is needed to provide additional and 
alternative methods of operating mobile communication terminal. 

1 0 SUMMARY OF THE INVENTION 

[0008] In accordance with one embodiment, a mobile communication terminal 
comprises a photographic apparatus connected to the terminal; an image processing unit for 
processing an image produced by the photographic apparatus, wherein control information is 
developed from the processed image; and an operational controlling unit for corresponding an 

1 5 operational function of the terminal to the control information. 

[0009] The image processing unit compares at least one initialization value with at 
least one corresponding value from the control information. The initialization value is set by a 
user. The image processing unit detects a first difference between the at least one initialization 
value and the at least one corresponding value. The control information comprises the first 

20 difference between the at least one initialization value and the at least one corresponding value 
processed from the image. In some embodiments, a user sets a first operational function of the 
terminal to correspond to the first difference. 

[0010] In another embodiment, a method for operating a mobile communication 
terminal with integrated photographic apparatus is provided. The method comprises 



3 



15-15 14:23 MC3N FROM: 



TD:213250B150 



PRGE:05 



photographing an image; processing the image for control information; setting an operational 
function of the mobile communication terminal to correspond to the control information; and 
operating the mobile communication terminal based on the control information. The step of 
processing the image comprises extracting a first value from the processed image; comparing the 
5 first value to an initialization value; determining a first difference between the first value and the 
initialization value; developing first control information derived from the first difference; and 
generating a control information signal based on the first control information. In some 
embodiments, the method further comprises photographing an image; extracting at least one 
value from the image; and setting the at least one value as the initialization value. 

1 0 1001 1] In accordance with yet another embodiment, a method for operating a mobile 

communication terminal with an integrated photographic apparatus is provided. The method 
comprises producing a first image from a first object with the photographic apparatus; detecting 
a diagnostic element within the first image; deriving at least a first value from the diagnostic 
element; deriving at least a first comprehensive value from the first value; determining a first 

1 5 difference between the first comprehensive value and a corresponding comprehensive 

initialization value derived from at least one initialization value: and assigning a first operational 
function of the mobile communication terminal to the first difference. 

[0012] The method may further comprise producing a second image from the first 
object with the photographic apparatus; detecting a diagnostic element within the second image; 

20 and deriving the at least one initialization value from the diagnostic element. At least one 
threshold value is applied to the comprehensive initialization value. The diagnostic element 
comprises a preliminary diagnostic element comprising a face featured on a head of an 
individual; and a secondary diagnostic element comprising a pair of eyes featured on the face of 
the individual. 
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(0013] Values are attributed to a first midpoint located between the eyes; a second 
midpoint located between a pair of shoulders; a vector drawn through the first and second 
midpoint; and an angle formed by the vector and a horizontal line joining the shoulders. The 
comprehensive initialization value comprises an approximate 90° angle formed by the vector and 
5 the horizontal line drawn joining the shoulders. In one embodiment, the comprehensive 
initialization value comprises a vector length measured when a horizontal line drawn joining 
eyes and containing the first midpoint is approximately parallel to the horizontal line joining the 
shoulders. 

[0014 ) These and other embodiments of the present invention will also become 
1 0 readily apparent to those ski lied in the art from the following detailed description of the 

embodiments having reference to the attached figures, the invention not being limited to any 
particular embodiments disclosed, 

BRIEF DESCRIPTION OF THE DRAWINGS 
15 (00 J 51 The accompanying drawings, which are included to provide a further 

understanding of the invention and are incorporated in and constitute a part of this specification, 
illustrate embodiments of the invention and, together with the description, serve to explain the 
principles of the invention. 

{0016 J FIGS. 1 A and IB illustrate conventional methods of operating a mobile 
20 communication terminal according to the related art; 

[0017] FIG. 2 illustrates a block diagram of an apparatus for operating a mobile 
communication terminal with an integrated photographic apparatus according to one 
embodiment of the present invention; 
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(0018J FIG, 3 illustrates a diagram of a method for operating a mobile 
communication terminal with an integrated photographic apparatus according to one 
embodiment of the present invention; 

[0019] FIG. 4 illustrates a flow chart of a process for initialization of a mobile 
communication terminal with an integrated photographic apparatus according to one 
embodiment of the present invention; and 

10020] FIGS. 5 A and 5B illustrate a flow chart of a process for movement detection 
by a mobile communication terminal with an integrated photographic apparatus according to one 
embodiment of the present invention. 

[0021] Features, elements, and aspects of the invention that are referenced by the 
same numerals in different figures represent the same, equivalent, or similar features, elements, 
or aspects in accordance with one or more embodiments. 

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS 
[0022] Reference will now be made in detail to one or more embodiments of the 
invention, examples of which are illustrated in the accompanying drawings, 

[0023] In FIG. 2, a block diagram of an apparatus for operating a mobile 
communication terminal with an integrated photographic apparatus, according to one 
embodiment of the present invention, is shown. The apparatus comprises an integrated 
photographic apparatus, such as a camera 10, mounted on the terminal. The camera 10 
reproduces an image on a display screen (not shown) of the terminal. An image processing unit 
20 installed on the terminal processes and analyzes images captured by the camera 10 to develop 
certain information. An operation controlling unit 30 detects the images and correlates 
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corresponding commands to the images, thereby providing to the us r operational functions for 
the terminal. 

(00241 T" e ca ™ era 10 i9 Preferably a digital camera including a sensor (not shown) to 
receive signals. The image processing unit 20, through the camera 10. receives image 
5 information including movements occurring in the image, such as an individual's movement of 
his lips, blinking of his eyes, or nodding of his head, for example. Such movements comprise a 
command control, wherein a command transmission, inputted by the user and received by the 
image processing unit 20, is correlated with the command control. A command transmission 
may comprise, for example, picture information, a particular alphanumeric character, or an icon. 
10 The operation controlling unit 30 detects the image processed by the image processing unit 20 
and, using a predetermined algorithm (discussed in detail below), programs the processed image 
to operate a function of the terminal, such as scrolling a menu, inputting an a selected 
alphanumeric character, or directing a cursor. 

[0025] Referring to FIG. 3. the process for initialising such a function is described 
1 5 herein. In FIG. 3, the image of a user's head is used as an example to operate directional control 
of a cursor within a terminal 's display. The image of the user's face is detected by the 
photographic apparatus integrated in the mobile communication terminal which is exemplified 
on the right as a folding-type mobile terminal. Furthermore, features of the user's face are also 
detected. Methods for detecting and distinguishing a user's facial features include a support 
20 vector machine (SVM) algorithm of pattern matching methods, a back-propagation neural 

network algorithm using normalized average face color information, and other suitable methods 
known to one of ordinary skill in the art. 

[0026] In using a SVM algorithm, an object having two categorical features is 
positioned in one-dimensional space, wherein the object is divided into one hyper-plane. The 
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proper and most suitable hypcr-planc is searched and determined to prevent misclasaification. 
With respect to a back-propagation neural network algorithm, a most suitable output value is 
derived from selected input values and a control weight. This algorithm is more appropriate for 
classifying and predicting based on a model. For example, a back-propagation neural network 

5 algorithm can be used to detect a studied facial shape by using a sliding window to view a black- 
and-white stationary image, wherein a plurality of facial images can be extracted. 

100271 FIG. 3, the face of an individual, such as a user, is photographed, wherein 

the facial region is detected from the photographed image. The individual's facial features are 
then separately processed. For example, the positions of the individual's eyes are detected by 

1 0 drawing a horizontal line 100 connecting the eyes. A vertical line 200 is drawn through a 

midpoint on line 100 between the individual's eyes and is extend below to serve as a Y-axis. A 
horizontal line 300 is drawn across the indiYiduars shoulders to provide as an X-axis, The 
intersection point of lines 200 and 300 serves an origin. 

|0028] Rotation of the individual's head, wherein the individual's face is moved in 

1 5 either the left or right directions, is measured by the angle 6. Right and left movements of the 
individual's face are measured based on reference point located between the individual's eyes. 
Alternatively, the angle formed by the tilting of the individual's head in either the left or right 
directions, in addition to either forwards or baokwards, can be measured. Accordingly, 
initialization values such as the lengths of lines 100 and 300, intersection points of lines 100 and 

20 200, and of 200 and 300, the distance between intersection points, and the angle 9 (initially 
approximately 90°), would be stored in the image processing unit 20, 

{0029] The flow chart of FIG. 4 illustrates the initialization process within the image 
processing unit 20 with respect to the present example. At step SI, an image is photographed by 
the photographic apparatus or camera 10 integrated in the mobile communication terminal. 
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When the image is inputted to the image processing unit 20, it is analyzed and processed for 
detectable information. For example, facial area within the image is determined in step S2. If a 
preliminary diagnostic element, such as a facial area, is not detected (step S3), then step SI is 
repeated, if a facial area or region is detected (step S4), then more detailed information is 
5 searched and detected in step S5, where eyes are detected, for example. Similar to step S3, if 
information such as the eyes region is not detected (step S6), then photography is repeated in 
step SI. 

[0030] After detecting the more detailed information (step S7), at least one 
initialization value is attributed to the detailed information. In the present example, three 
10 initialization values are attributed (steps S8-S10). For example, in referring to FIG. 3, a line 100 
of measured length is drawn across the distance between the eyes. Another initialization value is 
the perpendicular line 200. which is drawn through the midpoint of line 100. An additional 
initialization value is the measured line 300 that is drawn across the distance between the 
shoulders. 

15 |0031] In step Sll of FIG. 4, comprehensive initialization values, based upon the 

initialization values obtained in steps S8-S10, is generated. In the present example, the angle 8, 
which is formed by the vector (the intersection point of lines 200 and 300 to the intersection 
point of lines 100 and 200) and the either line 100 or line 200, represents a comprehensive 
initialization value. Furthermore, the length of the vector also represents a comprehensive 

20 initialization value. In step S 12, preset thresholds may optionally be applied to each 

initialization value to compensate for environmental conditions and unintentional movement by 
the user. For example, the initialization value for the angle 9 may be a range from approximately 
80° to approximately 100°. 
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[0032J Referring to FIG. 5A, an image is captured by a photographic apparatus 
mounted on the mobile communication terminal at step S100. When the image is inputted to the 
image processing unit 20, it is analyzed and processed for detectable information, such as fecial 
area within the image, in step SI 01. If the preliminary diagnostic element such as facial area is 
5 not detected in step S 1 02, then step S 1 00 is repeated. If a facial area or region is detected in step 
S103, then more detailed information, such as the region containing eyes, is searched and 
detected in step S104. Should the eyes region hot be detected in step S 1 05, then photography is 
repeated in step SI 00. 

[0033] After detecting the region containing the individual's eyes in step S106, a 
1 0 comparison value is attributed to the detailed information, where, in this example, a line of 
measured length is drawn across the distance between the eyes in step SI 07. Another 
comparison value, such as drawing a perpendicular line through the midpoint of line drawn in 
Step S107, is created in step SI 08. An additional comparison value is generated in step S109, 
where a measured line is drawn across the distance between the shoulders of the individual. In 
15 step S 1 1 0. comprehensive comparison values are obtained, such as by forming the angle 8' 

occurring between a vector and the line formed in step S109. and by measuring the length of the 
vector. The vector is formed from the intersecting points of the lines formed in steps S107-S109. 

[0034] Now referring to FIG. B and along direction arrow A, the angle 8' is measured 
and compared to the initialization value of angle 6 at step. If the angle 6' is determined to be less 
20 than the minimum threshold (i.e.. less than 80°) (step Sill), then the operational function 
corresponding to the sensed movement (e.g., tilting of the head to the left) is conducted (step 
S114). For example, movement of a cursor to the left within a menu displayed on the display 
screen would be shown. If the angle 8 1 is determined to be greater than the minimum threshold 
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(f.e., greater than 80°), then the 0 1 is determined whether to be greater than the maximum 
threshold (i.e., greater than 100°) (step S112). 

10035] Tf the angle 0' is determined not to be greater than the maximum threshold, 
then no angular and/or rotational movement is detected, wherein the process is reinitiated (step 
5 S 1 1 3 along directional arrow B). If the angle 8 1 is determined to be greater than the maximum 
threshold, then the operational function corresponding to the sensed movement (e.g., tilting of 
the head to the right) is conducted (step SI 14). According to the current example, the terminaPs 
screen would display a movement of the cursor to the right. 

(0036) The vector length is also determined for deviation from the range preset in the 
10 initialization value. In step SI 15, if the length is determined to be greater than the maximal 
initialization value, then the operational function corresponding to the sensed movement (e.g., 
nodding of the head upwards) is conducted (step Sll 8). In the present example, the operational 
function would consist of a cursor moving upwards in a display screen, tf the vector length is 
determined not to be greater than the maximal initialization value, then the vector length is 
1 5 determined whether to be less than the minimal initialization value (step S 1 1 6). 

[0037] If the vector length is determined not to be greater than the minimal 
initialization value, then no vertical movement is detected, wherein the process is reinitiated 
(step SI 17 along directional arrow C). If the vector length is determined to be less than the 
minimal initialization value, then the operational function corresponding to the sensed movement 
20 (e.g. y nodding of the head downwards) is conducted (step S 1 1 8). According to the present 
example, the terminal's screen would display a downwards movement of the cursor. 

[0038] In addition, to cursor control, the apparatus and method of the present 
invention, in one embodiment, can be applied to video games played on mobile terminals, 
wherein movements sensed and processed by the present invention can be translated to 
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movements of the game's protagonist. Additionally, separate movements, such as eye blinking 
and lip movement, may be incorporated into the algorithms programmed in the apparatus of the 
present invention to control addition functions of the terminal. In the gaming context, tilting the 
head to the right while blinking would be translated into the game's character moving to the right 
5 while firing a weapon, for example. In other operational contexts, repeated nodding of the head 
downwards can be translated into scrolling down a menu, wherein a blink may select a telephone 
number, for example. This technology may be incorporated into other applications which require 
user interaction with a text or graphic interface. 

[0039] The embodiments described above are to be considered in all aspects as 

10 illustrative only and not restrictive in any manner. Thus, other exemplary embodiments, system 
architectures, platforms, and implementations that can support various aspects of the invention 
may be utilized without departing from the essential characteristics described herein. These and 
various other adaptations and combinations of features of the embodiments disclosed are within 
the scope of the invention. The invention is defined by the claims and their full scope of 

15 equivalents. 
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